Towards Comparable Network Simulations

Simulations have been a valuable and much used tool in networking research for decades. New protocols are evaluated by simulations. Often, competing designs are judged by their respective performance in simulations. Despite this great importance the state-of-the-art in network simulations is nevertheless still low. A recent survey [6] showed that most publications in a top conference did not even give enough details to repeat the simulations. In this paper we go beyond repeatability and ask: Are different simulations comparable? We study various implementations of the IEEE 802.11 media access layer in ns-2 and OMNeT++ and report some dramatic differences. These findings indicate that two protocols cannot be compared meaningfully unless they are compared in the very same simulation environment. We claim that this problem limits the value of the respective publications because readers are forced to re-implement the work that is described in the paper rather than building on its results. Facing the additional problem that not all authors will agree on one simulator, we address ways of making different simulators comparable.